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Abstract 

In this work a linearly constrained minimization of a positive semidef- 
inite quadratic functional is examined. Our results are concerning infinite 
dimensional real Hilbert spaces, with a singular positive operator related 
to the functional, and considering as constraint a singular operator. The 
difference between the proposed minimization and previous work on this 
problem, is that it is considered for all vectors perpendicular to the kernel 
of the related operator or matrix. 
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1 Introduction 

The quadratic programming problem with equality constraints is one of the 
basic problems in optimization, in both the finite and the infinite dimensional 
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case. The general problem is to locate from within a given subset of a vector 
space the particular vector which minimizes a given functional. In this case, 
the subset of vectors is defined by a set of linear constraint relations and the 
functional is quadratic. In a classical book of Optimization Theory by Luen- 
berger [7], various similar optimization problems are presented, for both finite 
and infinite dimensions. 

In the field of applied mathematics, a strong interest is shown in applications 
of the generalized inverse of matrices or operators. Generalized inverses can 
be used whenever a matrix/ operator is singular, in many fields of both com- 
putational and also theoretical aspects. An application of the Moore-Penrose 
inverse in the finite dimensional case, is the minimization of a positive definite 
quadratic functional under linear constraints, presented in Manherz and Hakimi 
0. 

The problem studied in their work in the following: 

minimize f(x) = (x, Qx) + (p, x) + a,x G S 

where S — {x : Ax — b} and Q is a positive definite matrix. 

In this work we will extend this result for positive semidefinite matrices or 

operators acting on infinite dimensional real Hilbert spaces. Since in this case 

the operator studied is singular, the proposed minimization is attained for the 

vectors perpendicular to the kernel of the operator (or matrix) of the quadratic 

functional. 

2 Preliminaries and notation 

The notion of the generalized inverse of a matrix was first introduced by H. 
Moore in 1920, and again by R. Penrose in 1955. These two definitions are 
equivalent and the generalized inverse of an operator or matrix is also called the 
Moore- Penrose inverse. It is known that when T is singular, then its unique 
generalized inverse (known as the Moore- Penrose inverse) is defined. In 
the case when T is a real r x m matrix, Penrose showed that there is a unique 
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matrix satisfying the four Penrose equations, called the generalized inverse of T, 
noted by . The generalized inverse, known as Moore-Penrose inverse, of an 
bounded linear operator T with closed range, is the unique operator satisfying 
the following four conditions: 

TT^ = (TT^)* T^T = (T^T)* yyty — j 1 T^TT^ = T^ (1) 

where T* denotes the adjoint operator of T. 

In what follows, we consider TL a separable infinite dimensional Hilbert space, 
B(TL) denotes the set of all bounded operators on T~L and all operators mentioned 
are supposed to have closed range. In addition, 7t(T) will denote the range of 
an operator T, and Af(T) will denote its kernel. 

It is easy to see that TZ(T^) = ^/(T)- 1 , TT^ is the orthogonal projection of H 

onto TZ(T), denoted by Pp, and that T^T is the orthogonal projection of H onto 

AfiT) 1 - = K(T*) noted by P T ,. It is well known that K(T^) = K(T*). 

It is also known that is bounded if and only if T has a closed range. 

If T has a closed range and commutes with T\ then T is called an EP operator. 

EP operators constitute a wide class of operators which includes the self adjoint 

operators, the normal operators and the invertible operators. 

Let us consider the equation Tx = b, T G B(H), where T is singular. If b ^ R(T), 

then the equation has no solution. Therefore, instead of trying to solve the 

equation \\Tx — b\\ = 0, we may look for a vector u that minimizes the norm 

\\Tx — b\\. Note that the vector u is unique. In this case we consider the equation 

Tx = Pu(T)b, where Pr(t) is the orthogonal projection on 1Z(T). 

The following two propositions can be found in [5] and hold for operators and 

matrices: 

Proposition 2.1 Let T e M rxm and beW,b<f_ R{T). Then, for u E R m , the 

following are equivalent: 

(i) Tu = P R (r)b 

(ii) \\Tu-b\\ <|| Tx-b\\,Vx £R m 
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(Hi) T*Tu = T*b 

Let 1 = {u e W n \T*Tu = T*b}. This set of solutions is closed and convex, 
therefore, it has a unique vector with minimal norm. In the literature (eg. 
Groetsch [5]), B is known as the set of the generalized solutions. 

Proposition 2.2 Let T € C rxm and beC r ,b(£ R(T), and the equation Tx = 
b. Then, ifT> is the generalized inverse ofT, we have that T^b = u, where u is 
the minimal norm solution defined above. 

This property has an application in the problem of minimizing a symmetric 
positive definite quadratic functional subject to linear constraints, assumed con- 
sistent. 

As mentioned above, EP operators include normal and self adjoint operators, 
therefore the operator T in the quadratic form studied in this work is EP. An 
operator T with closed range is called EP if Af(T) = Af(T*). It is easy to see 
that 



TEPo K(T) = TZ(T*) <S> K(T) ®Af(T) = H^TT^ = T^T. 



(2) 



We take advantage of the fact that EP operators have a simple canonical form 
T = U 1 (A 1 © 0)C7 X * according to the decomposition U = K(T) ©jV(T). Indeed 

A 



an EP operator T has the following simple matrix form, T 







, where 



the form 



(see 0, 0] 



the operator A : 1Z(T) — > 1Z(T) is invertible, and its generalized inverse T* has 
A- 1 


In the finite dimensional case, if T G M ,ixn and rank(T) = r, the unitary matrix 
U has the form [Ui, U2) where U\ 6 R™ xr is an orthonormal basis for the column 
space of T and C/2 is an orthonormal basis for the null space of the matrix T. 
In Matzakos and Pappas [5] an algorithm is presented for the symbolic com- 
putation of the Moore-Penrose inverse for EP matrices and the corresponding 
factorization T = U(A 1 © 0)U* is presented. 

As mentioned above, a necessary condition for the existance of a bounded gen- 
eralized inverse is that the operator has closed range. Nevertheless, the range of 
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the product of two operators with closed range is not always closed. In Izumino 
[5] an equivalent condition is given, using orthogonal projections : 

Proposition 2.3 Let A and B be operators with closed range. Then, AB has 
closed range if and only if A^ABB^ has closed range. 

We will use the above theorem to prove the existence of the Moore- Penrose 
inverse used in our work. 

3 The Generalized inverse in linear constrained 
minimization 

The Generalized inverse of an operator or matrix plays a crucial role in many 
optimization problems where minimal norm solutions are studied. Ben-Israel 
and Greville (pQ chapter 3), and Campbell and Meyer ([2] chapter 3.6), have 
considered the constrained least- squares problem 

minimize \\Ax — b\\ under Bx = d 

with A, B, b, d, x all complex. 

In Chen [3], a similar problem is considered using partial ordering induced by 
cones, where the generalized inverse plays a fundamental role. 
The purpose of this paper is to extend the work done on this subject by Manherz 
and Hakimi [8 J. At first we will extend the finite dimensional results in the 
case of positive definite operators, and then we will examine the case when the 
operator is singular. 

3.1 Positive Definite Quadratic Forms 

Let Q be a positive definite symmetric matrix. The following theorem can be 
found in Manherz and Hakimi [5] : 
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Theorem 3.1 Let Q be positive definite. Consider the equation Ax = b . 
If the set S = {x : Ax = b} is not empty, the the problem : 

minimize = (x, Qx) + (p, x) + a,x G S 

and p, a arbitrary has the unique solution 

x = Q-i(AQ-^(±AQ-*p + b) - X -Q- X p 

A generalization of the above theorem for infinite dimensional Hilbert spaces, 
is by replacing Q with an invertible positive operator T. The operator A must 
be singular, otherwise this problem is trivial. In the case of operators instead 
of matrices, the proof is similar to Manherz and Hakimi [5], but the existence 
of a bounded Moore- Penrose inverse is not trivial like in the finite dimensional 
case. 

Lemma 3.2 Let T G Bill) be an invertible positive operator with closed range 
and A 6 singular with closed range. 

Then, the range of AT~^ is closed , where T~i denotes the operator (T^) -1 . 
Proof. : Using vrovosition \2.3\ we can see that 

AU(T-3)(T-i)t = Pa.(T-s)(T5) = P A . 

which has closed range. □ 

Therefore, the expression of the above theorem in infinite dimensional Hilbert 
spaces is the following: 

Theorem 3.3 Let T £ B(H) be positive definite and the equation Ax = b , 
where A is singular. 

If the set S — {x : Ax — b} is not empty, the the problem : 

minimize $(a;) = (x, Tx) + (p,x) + a,x G S 
with p,a G H arbitrary has the unique solution 

x = T-^{AT-^{^AT- 1 p + b) - -T- X p 
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Proof. : Since the existence of a bounded generalized inverse for the operator 
AT~i is proved is Lemma [3.21 the rest of the proof of the above theorem is 
similar to the finite dimensional case presented in [8] and is omitted. □ 

3.2 Positive Semidefinite Quadratic Forms 

An interesting case to examine, is when the positive operator T is singular, 
that is, T is positive semidefinite. In this case, since AT(T) ^ 0, we have that 
(x, Tx) = for all x G Af(T) and so, when Af(T) n S ^ this problem will take 
the form 

minimize $>(x) — (p, x) + a, for x 6 S fl N(T) 

which is usually treated using the simplex method, when the space is finite 
dimensional. 

A different approach in both the finite and infinite dimensional case would be to 
look among the vectors x G A/"(T)- L = 1Z(T*) = 1Z(T) for a minimizing vector 
for $>{x). In other words, we will look for the minimum of under the 

constraints Ax = b, x G 7Z(T). 

Using the fact that T is an EP operator, we will make use of the following 
proposition that can be found in Drivaliaris et al [4]: 

Proposition 3.4 Let T G B(T~L) with closed range. Then the following are 
equivalent: 

i) T is EP. 

ii) There exist Hilbert spaces K,\ and C±, U G £>(/Ci © unitary and A\ G 
£>(/Ci) isomorphism such that 

T = U(A 1 ®0)U*. 

We present a sketch of the proof for (1)=>(2): 

Proof. : Let /Ci = K(T), d = Af(T), U : K, x © d -> H with 

U(xi,x 2 ) = x\ + x 2 , 

for all xi G Tl(T) and x 2 G M{T), and A 1 = T\ n(T) : K{T) -)• TZ(T). Since T is 
EP, TZ(T) ©- 1 Af(T) = H and thus U is unitary. Moreover it is easy to see that 
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U*x = (PtX,Px(t)x), for all x G H. It is obvious that A\ is an isomorphism. 
A simple calculation shows that 

T = U(A 1 ®0)U*. 

□ It is easy to see that when T = U (A\ © 0)U* and T is positive, so is Ai, since 
(x,Tx) = (xi, Aixi),xi G TZ(T). 

In what follows, T will denote a singular positive operator with a canonical form 
T = U(Ai © 0)U* , R is the unique solution of the equation R 2 — A\ and 



x f = u 









R- 1 




u* = u 


















u* 



Theorem 3.5 Let T be positive semidefinite and X 2 = T. Consider the equa- 
tion Ax = b. 

If the set S = {x : Ax = b} is not empty, then the problem : 

minimize §(x) = (x, Tx) + (p, x) + a, x G S D AfiT) 1 - 
and p, a arbitrary has the unique solution 

x = X\AX^H^AT^p + b)-^p 
assuming that the operator Pa*Pt has closed range. 

Proof. : Since we will restrict the minimization for all vectors x £ M^) 1 - we 
have that (x,p) = (x,pi), where p\ = P-r(t)P for all vectors p £ H , according 
to the decomposition p = pi + p 2 G TZ{T) © Af(T). 

Let x,a,p G H with a and p arbitrary and not a function of T. Hence, the 
vector x that minimizes also minimizes 

tf(a;) = (Tx,x) + (x,p) + ^(Ttp,p) = (Tx,x) + (x, Pl ) + ^ Pl , Pl ) 

We can easily see that 

WXx+^p, f=*(x) 
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and that 

X = U(R®0)U*, = U{R- 1 ®Q)U* 

Set y = Xx+ \X^pt , which implies that y G K(X) = 1Z(T). 
We have that 

Xx = y- tj^Pi U{R 8 0)U*x = y- ^(R^ 1 © 0)U* Pl 

Hence, 

x = U(BT l © 0)U*y - \uiR~ 1 © 0)(iT 1 © 0)ETpi 
and so, since y,pi G 72.(T) 

i = X t y--T t pi, with x G 1Z(T^) = 1Z(T) = ^(T) 1 - 

Since Aa; = &, we have that AX^y = b + ^AT^pi and therefore, the minimal 
norm solution is 

y=(AXty(b+±Al*p 1 ) 
By substitution, we have that 

x = X\AX^)\^AT^ Pl + b) - ir1p x 

and since T^p = T^ Pl for all p G H =4- z = Xt(AXt)t(|ATtp + 6) - |Ttp 
The only thing that needs to be proved is the fact that the operator AX^ 
has closed range and so its Moore-Penrose Inverse is bounded. Since the two 
operators A and X' are arbitrary, one does not expect that the range of their 
product will always be closed. From Proposition 12 .3[ this is equivalent to the 
fact that the operator Pa*Pt has closed range because 

A t AX t (X t ) t = A^AX^X = A^AU(R~ 1 ®0)(R(B0)U* = P a *Pt 

and the proof is completed. □ In the sequel, we present an example which clarifies 
Theorem 13.51 In addition, the difference between the proposed minimization 
(x G TV^r)- 1 ) and the minimization for all x G H is clearly indicated. 
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Example 3.6 Let H = M , and the positive semidefinite matrix 



Q = 



26 10 -2 
10 8 2 
-2 2 2 



We are looking for the minimum of the functional 

f(u) = (u,Qu) + (p,u) + a, u <E Af(Q) ± n S 
with p — (1,2, 3) T , a = (0, 0, 0) T and the set of constraints S is defined as 
S = {(x, y, z) : 3x + y + z = —1}. 



The set Af(Q) ± has the form u — (2x — 3y, x, y) T , x,y gl. 

With easy computations, we can see that all vectors u E J\f(Q)' L satisfying the 



constraint Au — b, where A = 



3 1 1 



and b = —1, have the form 









u = { 


The matrices U, X^ are 




U = 


3 

VTo 



1 

V35 
7 


l 

vu 

7 




]_ 

V^LO 


3 
V35 


3 

Vu . 




= u 


' (V: 




3 





-8a; -3 -7x - 2 , 



A^ 1 




0.0667 -0.0623 
-0.063 0.1476 




U* 



0.1908 -0.0295 -0.0833 
-0.0295 0.2644 0.1861 
-0.0833 0.1861 0.1518 



Using theorem \3.5\ we can see that the minimizing vector of f(u) under 

{Au = b,ueAf{Q) ± } 



is 



= X\AX^{^AQ^p + b)- h$p = (-0.1019, -0.4369, -0.2573) T 
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The minimum value of f(u) is then equal to 1.4175 

In Figure 1 we can clearly see that the minimization of the functional f(u) for 
all vectors u £ Af(Q) having the form u = (2x — 3y,x,y),x,y £ R, belonging 
to the plane Zx + y + z = — 1, is attained. 

In Figure 2 we can see that among all vectors belonging to Af(Q) satisfy- 
ing Au — b, having the form u — (x, f , ~ 7 %~ 2 ) T \ x S R> the vector u = 
(-0.1019, -0.4369, -0.2573) T found from Theorem^ minimizes the functional 
/(«) ■ 



m- f(u) 

8» - 




x- coordinate 
y- coordinate 



Figure 1: Constrained minimization of f(u), u £ Af(Q) under Au = b 



We will also examine the minimization of f(u) in the cases when u £ M{Q) and 
when u £ R 3 , given that u £ S, so that the difference between them is clearly 
indicated: 

(i) When the minimization takes place for all vectors u £ Af{Q), then this 
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-1 -0,8 -0.6 -0.4 -02 02 0.4 0.6 0.B 1 

x- coordinate 

Figure 2: f(u), u e S H -/V(Q) 1 - 

problem takes the form : minimize f(u) = (p,u),u 6 S*. 

JFe can see iirci/i easy calculations, that the only vector u £ M(Q) n S* is 

u = (0.25, -0.5,0.75) T . In this case , f{u) = {p,u} = 1.5 

(it) When the minimization takes place for all vectors u £ M 3 ,u S S 1 , i/ien 
the minimizing vector of f(u) is w = (—0.1562,0.3958, — 0.9271) T and the 
minimum value of f(u) is f(w) — —1.8229. 

Corollary 3.7 We can easily see that, in the case when the vectors p,a are 
both equal to zero, the functional is a constrained quadratic form $(x) = (Tx,x) 
under Ax = b. 

In this case, the minimizing vector u belonging to Af(T) is then equal to u = 
X^AX'yb as it was discussed and proved in flOf - 

4 Conclusions 

In this work, we propose a constrained minimization in the case of a quadratic 
functional related to a positive semidefinite operator. The proposed minimiza- 
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tion takes place for all vectors perpendicular to the kernel of the corresponding 
operator. This proposed constrained minimization method has the advantage of 
a unique solution and is easy to implement. Practical importance of this result 
in the finite dimensional case, can be in numerous applications such as network 
analysis, filter design, spectral analysis, direction finding etc. In many of these 
cases, the knowledge of the non zero part of the solution corresponding to the 
matrix may be of importance. 
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